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Puipose: A low quality clinical trial could produce errors, and these errors could, in turn, distort the results of the clinical trial. 

To avoid applying distorted results of trials clinically, a quality analysis of clinical trials is needed. 

Materials and Methods: We selected randomized controlled trials (RCTs) about erectile dysfunction (ED) conducted in Korea 
using Medline and KoreaMed. Quality assessment of selected RCTs was performed using three assessment tools Qadad scales, 
van Tulder scale, Cochrane Collaboration Risk of Bias Tool [CCRBT]). 

Results: The first RCT about ED conducted in Korea was published in 2002. Since 2002, a total of 20 RCTs have been published 
in medical journals. Among the 20 articles, only 1 article was found to have a low risk of bias according to the CCRBT. On the 
Jadad scale, there were 1 7 high quality articles, while 1 9 articles were assessed as high quality by the VTS. Only 2 RCTs described 
the randomization method adequately. Only 1 RCT presented allocation concealment. 

Conclusions: A low quality clinical trial could produce errors, and these errors could, in turn, distort the results of the clinical 
trial. To avoid applying distorted results of trials clinically, a quality analysis of clinical trials is needed. The quality of RCTs was 
found to be high because almost all of the selected RCTs were double blinded studies. However, the quality of RCTs was 
inadequate with regard to the lack of randomization and absence of allocation concealment. Therefore, performing adequate 
randomization and adding a description of the appropriate concealment of allocation may improve the quality of RCTs. 
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INTRODUCTION 

Among the many designs for clinical trials, the random- 
ized controlled trial (RCT) shows the highest level of evi- 
dence, which can reduce the bias most effectively. The re- 
sults of RCTs are typically used as the basis for clinical 
application.^ Moreover, according to the increasing im- 
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portance of evidence-based medicine, the importance of 
RCTs has drawn even more attention.^ However, even 
RCTs cannot eliminate all bias. Bias can occur at any of the 
steps in conducting trials and can lead to incorrect results.^ 
These incorrect results can harm patients and cause un- 
necessary medical expenses."* There are some methods for 
preventing the use of incorrect information in clinical 
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applications. Representatively, peer review is one of the 
methods by which articles are assessed before 
publication.^ Another method involves the methodo- 
logical quality assessment of articles, in which the design, 
conduct, and analysis of clinical trials is evaluated.* 
Assessment of the methodological quality of articles repre- 
sents the overall quality of clinical trials.'' Methodological 
quality assessment of clinical trials can include the use of 
scales, individual markers, and checklists.^ Scales address 
randomization, double blinding, and dropout rates. The 
Jadad Quality Assessment Scale (Jadad scale) is the assess- 
ment tool that can assess the scales.^ Allocation conceal- 
ment, as an individual marker, is an index that can assess 
the randomized allocation sequence. Allocation conceal- 
ment is essential to avoid selection bias.^ The Jadad scale 
has been widely used because of its simplicity. However, 
the Jadad scale cannot evaluate individual markers. On 
the other hand, the van Tulder scale (VTS) and Cochrane 
Collaboration Risk of Bias Tool (CCRBT) include indices of 
individual markers. 

Erectile dysfunction (ED) is an inability to achieve an 
erection sufficient to enable sexual intercourse. Many fac- 
tors are associated with ED, including vascular, psycho- 
logical, hormonal, and neurological factors.^" ED affects 
150 million males worldwide, and the prevalence of ED is 
expected to more than double within 1 0 years due to the 
increasing lifespan." In Korea, some clinical trials about 
ED have been performed. In the present study, to suggest 
a direction for clinical trials about ED, we performed a 
methodological quality assessment of RCTs about ED. 

iVtATERIALS AND METHODS 

1. Data collection 

Data collection was done by using a Medline and 
KoreaMed search. Medical Subject Headings (MeSH) 
phrases related to ED were used for searching. There were 
no limitations on the language. In total, 20 RCTs were 
identified in the search. 

2. Assessment of the quality of RCTs 

The Jadad scale, the VTS, and the CCRBT were used for 
quality assessment. The quality assessment of RCTs was 
conducted according to the publication year, while the 



quality assessment was conducted by type of intervention, 
presence of funding, and whether it was reviewed by an 
institutional review board (IRB). 

1) Jadad scale 

The Jadad scale is composed of five points: two in rela- 
tion to randomization, two in relation to blinding, and one 
in relation to the drop-out rate.^ When the RCT includes 
only general comments without a detailed description of 
randomization and blinding, one point for each index is 
given. One point is added when there is a detailed and ap- 
propriate description. However, one point is deducted 
when the description is inappropriate. When the specified 
number and reasons for drop-outs by each subject group 
are provided, one point is given. Even if there are no 
drop-outs, this should be stated specifically. When the to- 
tal is ^3 points, it is assessed as high quality but when it 
is -^2 points, it is assessed as low quality. However, if it 
was not possible for the design of the study to be double 
blinded, it is assessed as high quality when the total score 
is ^2 points. 

2) van Tulder scale 

The VTS is designed to assess 11 components 
(randomization, allocation concealment, baseline charac- 
teristics, patient blinding, caregiver blinding, observer 
blinding, co-intervention, compliance, drop-out rate, 
end-point assessment time point, and intention-to-treat 
analysis). Its assessment method is to select 'yes', 'no', or 
'don't know' for each item, and when ^5 items are sat- 
isfied (^5 points), the quality of the report is deemed high. 

3) Cochrane collaboration risk of bias tool 

The CCRBT assesses the quality of RCTs in six classi- 
fications (sequence generation, allocation concealment, 
blinding, incomplete outcome data, selective outcome re- 
porting, and other potential threats to validity). The assess- 
ment indicates 'yes', 'no', or 'unclear' for each domain. In 
cases where the first three questions are answered with 
'yes' and when no important concerns related to the last 
three domains are identified, it is classified as having a low 
risk of bias, while cases where it is assessed in <2 domains 
with 'unclear' or 'no', it is classified as having a moderate 
risk of bias. The cases assessed in ^3 domains with 
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'unclear' or 'no' are classified as having a high risk of bias. 
3. Statistical analysis 

The Kruskal-Wallis test was used to compare and analyze 
the respective scores obtained by each assessment tool, 
while a chi-squared test was used to compare and analyze 
the ratio of the high quality articles and the quality assess- 
ment outcomes from the CCRBT. The quality assessment of 
RCTs according to the publication year was analyzed by 
one-way ANOVA analysis. PASW Statistics 18.0 (IBM Co., 
Armonk, NY, USA) was used for all statistical analyses, and 
apvalueof <0.05 was considered statistically significant. 

Table 1. Characteristics of RCTs 



Characteristic 



Value (n = 20) 



Intervention 

Drug 

Non-drug 
Adequate randomization 
Allocation concealment 
Double blinding 
IRB 

Funding 
SCI/ SCIE 



20 (100.00) 

0 (0.00) 
2 (10.00) 

1 (5.00) 
19 (95.00) 
18 (90.00) 

15 (75.00) 

16 (80.00) 



Values are presented as number (%). 
RCTs: randomized controlled trials, IRB; institutional review 
board, SCI: science citation index, SCIE: science citation 
index expanded. 



RESULTS 

A total of 20 RCTs were selected in an electronic search. 
All of the selected RCTs were conducted to evaluate drugs. 
Among the 20 RCTs, 19 RCTs were double-blind studies, 
and there were only 2 RCTs which described the random- 
ization method adequately. Only 1 RCT presented alloca- 
tion concealment. Eighteen RCTs mentioned review by an 
IRB and 15 RCTs were funded studies. Sixteen RCTs had 
been published in Science Citation Index (SCI) or Science 
Citation Index Expanded (SCIE) journals (Table 1). 

The largest number of articles had been conducted to 
evaluate mirodenafil (n =6), and 5 articles evaluated ude- 
nafil (Table 2). 

Among the 20 articles, only 1 article was assessed as 
having a low risk of bias, 3 articles were assessed as having 



Table 2. Drugs evaluated in the included studies 
Drug n 



Mirodenafil 


6 


Udenafil 


5 


Ginseng extract 


4 


Sildenafil 


2 


Tadaiafi! 


1 


Avanafil 


1 


Herbal formula 


1 


Total 


20 



Table 3. Quality assessment of RCTs according to publication year 

Jadad scale van Tulder scale 



Year RCT i_|j„h 

Score ,. p vakie Score 

qLiality 



2002 


1 


3 


1 (100.00) 


7 


2003 


1 


3 


1 (100.00) 


8 


2006 


2 


3 


1 (50.00) 


7.5 


2007 


1 


2 


0 (0.00) 


6 


2008 


4 


2.5 


3 (75.00) 


7 


2009 


3 


3 


3 (100) 


0.478 8 


2010 


3 


3 


3 (100) 


8 


2011 


2 


3 


2 (100) 


8 


2012 


2 


3 


2 (100) 


7.5 


2013 


1 


5 


1 (100) 


9 


Total 


20 


2.95 ± 0.76 1 7 (85.00) 


7.60±1.14 



Cochrane's assessment of risk of bias 



High . 

?. p value 
quality 


High 

risk 


Moderate 
risk 


Low 

risk 


1 (100.00) 


0 


1 


0 


1 (100.00) 


0 


1 


0 


2 (100.00) 


1 


1 


0 


1 (100.00) 


1 


0 


0 


3 (75.00) 


1 


3 


0 


3 (100.00) 0.474 


0 


3 


0 


3 (100.00) 


0 


3 


0 


2 (100.00) 


0 


2 


0 


2 (100.00) 


0 


2 


0 


1 (100.00) 


0 


0 


1 


19 (95.00) 


3 


16 


1 



p value 



Values are presented as number (%) or mean + standard deviation. 
RCT: randomized controlled trial. 
Kruskal-Wallis test, ^chi-squared test. 



The World Journal of 

Men's Health 



1 60 World J Mens Health Vol. 31 , No. 2, August 201 3 



University of Ulsan 
Korea University 
Hanyang University 
Gyeongsang National University 
Kyung Hee University 
Pusan National University 
Yonsei University 
Chonbul< National University 
Seoul National University 




0 1 2 3 4 5 6 7 
Number of publicated articles 



Fig. 1. Medical institution of corresponding author. 



a moderate risk of bias, and 1 6 articles were evaluated as 
having a high risk of bias by the CCRBT. In the Jadad scale, 
there were 17 high quality articles, while 1 9 articles were 
assessed as high quality articles by the VTS. In the Jadad 
scale and VTS, there was no statistical improvement ac- 
cording to the publication year. However, in the CCRBT, 
there was a statistical difference in quality according to the 
publication year (p =0.036) (Table 3). 

Sixteen RCTs had been conducted at Seoul National 
University. Chonbuk National University and Yonsei 
University had each published three RCTs. Kyung Hee 
University and Pusan National University had conducted 
two RCTs each (Fig. 1). 

Three RCTs had been published in The Korean Journal 
of Urology, in Asian Journal of Andrology and in British 
Journal of Urology International. And, 2 RCTs had pub- 
lished in Journal of Urology. 

DISCUSSION 

In the present study, the quality of published RCTs 
about ED conducted in Korean medical institutions was 
assessed. In the Medline database, about 900 RCTs that 
dealt with ED were found. However, there were only 20 
RCTs about ED that had been conducted in Korea. Only 
one RCT was assessed as having a low risk of bias accord- 
ing to the CCRBT. On the Jadad scale and the VTS, there 
was no significant difference in the quality of RCTs accord- 
ing to the publication year. 

ED is defined as the inability to attain or maintain a pen- 



ile erection sufficient for successful vaginal intercourse.^^ 
In a landmark population-based study, the crude in- 
cidence of ED was 26 per 1 ,000 man-years. Many factors 
such as vascular, psychological, neurological and hormo- 
nal abnormalities are associated with ED.^^ Many ther- 
apeutic strategies have been introduced, such as psycho- 
logical therapy, lifestyle modification, phosphodiesterase 
type 5 inhibitors (PDE5-ls), testosterone, and penile 
prostheses.'^ Notably, PDE5-ls have been used as a first 
line of treatment for ED since sildenafil was introduced in 
1 998.'''''^ Even though many treatment options have been 
proposed for ED, there are many limitations in the man- 
agement of ED.'^ Because of the high incidence of ED and 
therapeutic limitations, many studies have been con- 
ducted about ED. However, none have assessed the qual- 
ity of RCTs about ED. 

Some previous analyses have been performed on the 
quality of urological RCTs. Lee et al" analyzed the quality 
of RCTs that had been published in the Korean journal of 
Urology by the Jadad scale. Twenty-eight RCTs have ap- 
peared since 1991 in the Korean Journal of Urology. The 
mean Jadad scale score of those RCTs was 1 .75. In addi- 
tion, 8 RCTs were assessed as high quality. Moreover, Lee 
et al'' showed that only one article had adequate alloca- 
tion concealment. In the present study, compared to Lee et 
al,'^ the mean Jadad scale score (2.95 + 0.76) was higher. 
However, in the present study, only 1 RCT had adequate 
allocation concealment. According to Schuiz and 
Crimes,^" the absence of concealment of allocation can 
damage the randomization in the process of the study, 
even if the randomization is well conducted. This may dis- 
tort the results by up to 40%. Hewitt et al^' reported that 
about 46% of RCTs published in four journals (British 
Medical Journal, Journal of the American Medical 
Association, Lancet, and New England Journal of 
Medicine) described inappropriate allocation conceal- 
ment or failed to describe it clearly. Appropriate study de- 
sign and presentation of allocation concealment should 
improve the quality of RCTs. 

Chung et aP assessed the quality of RCTs published in 
the Journal of Korean Medical Science. They reported that 
the Jadad scale score was 2.30 ± 1 .39 and the VTS score 
was 4.98±2.1 8. According to the CCRBT, 68.18% of the 
RCTs were assessed as having a high risk of bias, 20.45% 
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of the RCTs were assessed as having a moderate risk of 
bias, and 1 1 .36% of the RCTs were assessed as having a 
low risk of bias. Compared to the previous two studies, the 
quality of RCTs in the present study seemed to be higher 
according to the Jadad scale and VTS. This may be because 
almost all of the selected RCTs in the present study were 
drug studies. In studies that use a drug as the intervention, 
double blinding is easier than in non-drug studies because 
of to the possibility of using a placebo. 

The present study showed that there were differences in 
the quality analysis of RCTs according to the three differ- 
ent tools. There is no consensus on what are considered 
highly accurate and valid quality assessment tools.^ The 
Jadad scale has simple quality assessment questions, but it 
does not include assessment of individual markers. 
Therefore, additional quality analyses were performed us- 
ing the VTS and CCRBT to supplement in this regard. 

The review from an IRB could serve as an acknowledge- 
ment ofthe feasibility of a study. IRB review should bean 
important factor in the improvement of the quality of 
RCTs.^ Funding of clinical trials should enable the estab- 
lishment of a well-designed study and performance of 
well-organized research, resulting in many high quality 
articles.^'^'" In the present study, almost all of the studies 
were funded study and had been reviewed by an IRB. This 
may be an one of the factors that improved the quality of 
the RCTs. The relatively small numbers of selected RCTs 
enabled the comparison of quality between funded stud- 
ies and non-funded studies. It also enabled the compar- 
ison of quality according to whether the IRB reviewed the 
study or not. 

The limitation ofthe present study is the small number 
of RCTs included. In addition, the assessment of quality 
was conducted by one researcher. This could have in- 
troduced bias because the selection of RCTs and quality 
assessment depends on the subjective judgment of the 
researcher. 

CONCLUSIONS 

The assessed number of RCTs about ED that have been 
conducted in Korea is small. Even though the quality of 
RCTs in the present study was superior to that of other pre- 
vious studies, considering that almost all ofthe RCTs in- 



cluded in the present study were double blinded, we can- 
not conclude that the RCTs in the present study are superi- 
or to others. In any case, our results emphasize that uro- 
logical researchers should focus on performing high qual- 
ity studies that have been reviewed by an IRB, performing 
adequate randomization, and describing the appropriate 
concealment of allocation. 
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